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By Alyssa M. Walters, Patrick C. Kyllonen and Janice W. Plante 


Developing a Standardized 
Letter of Recommendation 


Abstract 

The Standardized Letter of Recommendation (SLR) is a Web-based admission tool de- 
signed to replace traditional, narrative letters of recommendation with a more systematic 
and equitable source of information about applicants to institutions of higher education. 
The SLR includes a rating scale and open-ended response space that prompt evaluators 
to describe applicants on a set of qualities identified by faculty in previous research as 
important to academic success, including knowledge and skills, creativity, communica- 
tion skills, motivation, self-organization, professionalism and maturity, and teamwork. 
Admission committees, in turn, receive a Web-based, interactive report of the applicant 
information. Researchers asked 421 graduate faculty and administrator respondents to 
use the SLR and then report their interest in and preferences for the form, content and 
function of an SLR. Overall, a majority of faculty and administrators preferred the SLR to 
the system currently being used at their institutions. The researchers list several practical 
implications that adopting institutions may wish to consider, and provide recommenda- 
tions for subsequent activities to develop the project. 


Research to Develop a Standardized Letter of Recommendation 

Admission committees currently rely on letters of recommenda- 
tion as important sources of qualitative information about ap- 
plicants to institutions of higher education {Briel, Bejar, Chan- 
dler, Powell, Manning, Robinson, Smallwood, Vitella, and Welsh, 
2000; Powers and Fowles, 2000; Walpole, Burton, Kanyi, and 
Jackenthal, 2001). Letters of recommendation allow committees 
to glean valued information about students such as motivation, 
persistence, creativity, and personality that are otherwise diffi- 
cult to obtain from traditional admission tools. Moreover, surveys 
of psychology graduate admission committees have found that 
letters of recommendation are one of the top three criteria for 
admission of students, along with undergraduate Grade Point 
Average (GPA) and Graduate Record Examination (GRE) scores 
{Keith-Spiegel, Tabachnick, and Spiegel, 1994; Landrum, Je- 
glum, and Cashin, 1994). 


However, there are several limitations to the reliance on 
these letters that may impair their usefulness and fairness as a 
central component of the admission process. First, letters often 
contain overly general and vague language, and admission com- 
mittees report having to “read between the lines” to determine 
the letter writer’s true intentions (Walpole et al., 2001). Not sur- 
prisingly, letters with specific examples of applicants' qualities 
stand out and are considered more powerful (Knouse, 1983). 
Therefore, the information contained in letters is often left to the 
subjective interpretation of the reader, leading to misinterpreta- 
tions or mistakes. 

Secondly, letters of recommendation lack standardiza- 
tion — one evaluator’s “extremely hard worker” could be another’s 
“typical student.” Thus, whether or not a letter includes valid 
evidence for qualitative variables such as motivation and team- 
work will often depend on the evaluator’s standards, letter-writing 
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skills, and knowledge of the admission office agenda. Finally, 
lefters of recommendation are time intensive for bofh the letter 
writers and letter readers. Writers must recall an appropriate set 
of experiences and qualifies on which fo base fheir evaluafions. 
As nofed previously, readers must attend carefully fo both explicit 
and implicit information about the applicant in order to interpret 
the evaluation. These limitations reduce the inherent value of the 
letter of recommendation for admission commiftees. 

This article outlines the development of a Standardized 
Letter of Recommendafion (SLR) to address these limitations. 
Maintaining a standard language, set of concepts, and response 
options could remove much of fhe ambiguity and the need for 
subjective interpretation and also provide a more efficient for- 
mat. Standardization could also reduce the importance of letter 
writing ability and savvy about the letter writing process that cur- 
rently exists in conventional letters of recommendafion. 

The Standardized Letter of Recommendation 

The researchers designed the SLR as a Web-based admission 
tool to supplement or replace existing letters of recommenda- 
fion for graduate school admission. The hope was that the SLR 
would provide a more valid and systematic source of informafion 
abouf applicant qualities than current letters of recommenda- 
fion. The SLR direcfed an evaluafor to describe an applicant on 
seven qualities: (a) knowledge and skills, (b) creativity, (c) com- 
munication skills, (d) motivation, (e) self-organization, (f) profes- 
sionalism and mafurify, and (g) feamwork. For each quality, the 
evaluator is asked to rate the candidate on four specific items 
{e.g., for “creativity," an item is “produces novel ideas”) on a 
five-point scale (“below average” to “truly exceptional” with a 
“don’t know” option). The evaluator was also allowed to elabo- 
rate on or provide examples of that quality in a comment box. In 
this way, evaluators who prefer written text to a rating system will 
not be constrained by the SLR. 

The SLR was paired with an interactive Web-based graphical 
summary of fhe evaluations of each applicanf designed for use by 
an admission commiftee. The summary provided individual and 
aggregated evaluator responses on each quality. It also enabled 
additional comparisons, such as with other applicants, or with 
reference groups, such as all admitfed sfudents, or all applicants 
across the country. 

Similar to standardized tests, the SLR yielded stable popu- 
lation norms, facilitating candidate evaluation. Standardization 
allowed for comparisons across a variety of factors, such as eth- 
nicity and gender. Further, the SLR created an on-going data col- 
lection effort, which will enable validation against a wide range 
of educafional outcomes. Continued research ensured that the 
constructs included on an SLR represented those most valued 
by admission committees. In these ways, the SLR can generate 
a feedback loop of student data, which will inform education 
practice, policy, and retention efforts. 


Previous Research on Non-Cognitive Skills and Graduate Admission 

The SLR was designed to assess the qualities of most interest 
to institutions. The SLR that was designed for graduate admis- 
sion assessed a set of qualities derived from a series of GRE- 
sponsored projects that have attempted to identify the factors 
that faculty members consider important for success in graduate 
school. The Florizons initiative staff (Briel, et al., 2000) inter- 
viewed 71 faculty members, graduate directors and deans from 
a variety of departments at 14 U.S. institutions. When asked to 
describe factors important for admission, interviewees consis- 
tently mentioned non-cognitive factors, such as motivation, per- 
sistence, teamwork, and research initiative. This led the authors 
to recommend that a standardized assessment of key non-cogni- 
five skills be developed. 

In a follow-up to Florizons, Walpole et al. (2001) conducted 
open-ended telephone interviews with 16 faculty members and 
five deans. Interviewees consistently mentioned motivation, cu- 
riosity, persistence, goal orientation, communication skills, orga- 
nizational skills, and the ability to multi-task as important admis- 
sion criteria. Given the importance of fhese facfors fo admission 
decisions and the relative difficulty of extracfing this informa- 
tion from applicanf folders, interviewees expressed a desire for a 
means to measure and present information on student attributes. 
Interviewees believed that such a service might minimize attri- 
tion rates and time to degree. 

Additional GRE-sponsored research has furthered understand- 
ing of fhe consfructs that graduate admission committees hope to 
learn from letters of recommendafion. Powers and Fowles (2000) 
asked nine psychology faculty members and 14 history faculty 
members to review simulated applicant admission folders. The 
folders confalned quantified ratings of informafion contained in 
each applicant’s letters of recommendafion and personal essays. 
In addifion to rating the extent to which they would recommend 
admitting the student, participants indicated the importance of 
several non-cognifive factors contained in the letters and essays. 
While intellectual ability and research aptitude were the two most 
important variables gleaned from leffers of recommendation, sev- 
eral “soft skills” (including communication skills, creativity, inde- 
pendentthinking, reliability, initiative, character, emotional matu- 
rity, and motivation) were also rated as highly important. 

Past research has demonstrated both interest in, as well as 
some direction for fhe content of a non-cognitive supplement to 
current admission materials (Briel et al, 2000; Powers and Fowles, 
2000; Walpole et al, 2001). Currently, some colleges and univer- 
sities present their own versions of recommendafion forms. Flow- 
ever, fhese institution-specific forms have fwo limitafions. First, 
content has typically been chosen informally, withouf a document- 
ed research base. Second, their use precludes some of fhe ben- 
efits available with a common, widely adopted standardized letter 
of recommendafion, such as stable national and applicant school 
norms that could be used for evaluafing applicanfs’ scores. 
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“Information gleaned 
from the focus groups 
was used to develop a 
comprehensive telephone 
survey that was designed 
to evaluate faculty 
reactions to completing 
the SLR Web site 
prototype.” 


Overview of Research to Develop the SLR for Graduate Admission 

In an effort to develop a viable and effective admission tool, research- 
ers surveyed graduate faculty and administrators who would be po- 
tential users of the SLR. The research proceeded in two steps. Pre- 
liminary research included focus groups to determine the preferred 
format and function of an SLR prototype. In the primary research 
initiative, participants interacted with the prototype that emerged 
from the preliminary research and described their preferences and 
overall reactions to the SLR via an extensive telephone survey. 

Preliminary Research to Develop an SLR Web site Prototype 

A necessary first step was to conduct focus groups with a sample 
of potential SLR users to inform the development of a working SLR 
Web site prototype. Four focus groups were conducted with gradu- 
ate faculty and department chairs at colleges and universities in 
Boston, Atlanta, Chicago, and Los Angeles. Participants were 38 
faculty members representing institutions that varied according to 
key factors such as size, public versus private, and terminal mas- 
ters versus doctoral level programs. Researchers included Flistori- 
cally Black Colleges and Universities (FIBCU's) and programs that 
cater to international students. Participants came from a broad 
array of disciplines that fell into five areas: natural sciences, engi- 
neering, social sciences, humanities and arts, and education. 

The qualities that were deemed important to faculty partici- 
pants in prior studies were used to develop the content of the 
SLR prototype. Using an a priori set of well-esfablished quali- 
fies (i.e., Kyllonen, Walters, and Kaufman, 2004) as a frame- 
work, these skills were grouped into a set of seven core qualifies: 
knowledge and skills, creativity, communication skills, motiva- 
tion, self-organization, professionalism and mafurify, and team- 
work. Participants evaluated paper versions of five sample evalu- 
ator forms and five sample reporf forms for an SLR and discussed 
ifs overall purpose and potential usefulness. 


The marketing team prepared a comprehensive final reporf, 
summarizing and analyzing the findings and recommendations 
from the focus groups {Cohn Research, 2002). Participants ex- 
pressed a wide range of preferences for the features and function- 
ality of an SLR. We then developed a working Web site prototype 
to accommodate this range of preferences. Web delivery allowed 
the accommodation of divergent preferences — facuify preferring 
writfen text needed simply to click on a comments box, while 
faculty preferring numeric ratings could use graphs and norma- 
tive data.^ The input and output forms that received the highest 
rankings by focus group members became the basis for the site 
format. For the final Web site, each quality was defined by a 
preliminary sef of four items, which were developed in collabora- 
tion with Lewis Goldberg, a leading personality psychologist who 
served as a consultant for fhe project. 

Primary Survey Research 

The primary study included a telephone survey with a large, 
representative sample of graduate faculty and administrators to 
accomplish the following objectives: (1) Obtain feedback from 
graduafe institufions about the value and usefulness of the SLR 
and gather recommendations for improvements to a prototype 
of the SLR; and (2) evaluate issues related to the likelihood of 
adopfion of the SLR in the admission process. 

Method 

Information gleaned from the focus groups was used to develop 
a comprehensive telephone survey that was designed to evaluate 
faculty reactions to completing the SLR Web site prototype. 

Participants 

The telephone survey was conducted with 421 institutional rep- 
resentatives. The institutions included in the study are a repre- 
sentative sample of currenf GRE score recipients. To ensure that 
the appropriate types of respondenfs were selected for the study, 
GRE program direction did some advance investigation with in- 
stitutional contacts to determine the appropriate level and mix 
of sfudy parficipants. The selected participants represented the 
range of programs and institutions outlined in the focus groups. 
Individuals who provide input into graduate admission policy 
and/or make admission decisions participated in the study. This 
includes graduate faculty (n = 349), faculty from FIBCU's (n = 
33), and graduafe school administrators (n = 39). For the gradu- 
ate faculty sample, the median number of letters written per year 
was 10, and the median number of letters read per year was 60. 
For the FIBCU faculty sample, the median number of lefters wrif- 
ten per year was 12.^ 


'Seven GRE Board members participated in a usability study of an initial version 
of the Web site. Each participant was assigned one observer who asked a series of 
scripted questions and recorded all feedback and observations. Several changes 
were made to the Web site In response to this feedback. 

^ Most HBCU faculty did not read letters because they represented undergraduate 
Institutions. 
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Procedure 

The marketing team developed a survey using input from both 
research staff and the GRE program staff. Institutional respon- 
dents were contacted by telephone and screened to determine 
their eligibility and interest in participating in the study. Con- 
firmed study participants received a Word file via email that con- 
tained the Web site information and a confidentiality statement 
for signature. Participants were given approximately one week to 
view the Web site and be available for the telephone interview. 
Participants were instructed to complete the SLR as if evaluating 
the last applicant for whom they wrote a letter of recommen- 
dation. On the scheduled date the interviewer contacted each 
participant and administered a 20-30 minute survey. The survey 
included questions about the input and output forms, as well as 
some additional questions that will help evaluate the viability of 
the SLR concept. 

Results 

The marketing team prepared a comprehensive report of survey 
responses (Cohn Research, 2003). Below are the frequencies 
and percentages for all major findings. 


Reactions to the SLR 

Using a one to five scale with anchors at much better and much 
worse, 59 percent of the graduate faculty sample rated the SLR 
as somewhat better (37 percent) or much better (22 percent) 
than traditional letters of recommendation (11 percent rated it 
as somewhat worse and 4 percent rated it as much worse). Re- 
sponses did not vary significantly as a function of discipline. 
Similar patterns emerged for both the HBCU faculty and admin- 
istrator samples^. (See Figure 1). Across all samples, the most 
common reason provided for preferring the SLR was that the 
“Detailed list of qualities is more thorough and stimulates the 
writer” (the most common reason for a negative evaluation was 
“Prefer a personalized and tailored letter”). Appendix A lists the 
top reasons provided for using each response option. 

Figure 2 describes participants’ reactions to specific fea- 
tures of the SLR (using a one to five scale with anchors at 
strongly agree and strongly disagree). A majority of respondents 
strongly agreed or somewhat agreed that in comparison to nar- 
rative letters of recommendafion, the SLR includes the quali- 
ties that are important to success, results in fairer comparisons, 


Figure 1: 

Responses to the 
question: Do you think 
the SLR is much better, 
somewhat better, about 
the same, somewhat 
worse, or much worse 
than the letter forms you 
typically use for graduate 
school applicants? 


Graduate Faculty 


HBCU Faculty 


Administrators 





■ Much better 

□ Somewhat better 

□ About the same 

□ Somewhat worse 

□ Much worse 

■ Do not know 



0% 10% 20% 30% 40% 50% 60% 


3 Examination of responses revealed that the three samples varied little. However, 
we did not conduct formal statisfical comparisons across groups. 
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is quicker than sending separate letters to multiple schools, is 
easier to complete, obtains more useful information, and is more 
convenient. Also, 73 percent of the respondents strongly agreed 
(34 percent) or somewhat agreed {39 percent) that rating the 
candidate on four items per quality would make the SLR a more 
reliable assessment. 

When prompted to suggest revisions to the SLR, the top four 
suggestions were: (a) the SLR should be shortened (seven percent 
of all respondents); (b) the response scale needs to be more clearly 
aligned with the questions (five percent); (c) Add questions about 
specific experiences and knowledge of the field (four percent); and 
(d) Programs should be able to customize the forms (four percent). 


Reactions to the SLR Report 

Sixty-two percent of the graduate faculty sample rated the SLR 
report as much better (32 percent) or somewhat better (30 per- 
cent) than the format their graduate school currently uses to 
analyze applicants' letters of recommendation. (See Figure 3). 
Similarly, 69 percent of the administrator sample rated the SLR 
report as much better (33 percent) or somewhat better {36 per- 
cent). The top reasons participants used each response option 
are described in Appendix B. 


Figure 2: 
Responses to questions 
about specific features 
of the SLR 


Use of four ifems is more reliable 


Qualifies are most important to success 


SLR results in fairer comparisons 


SLR is quicker (one for all schools) 


SLR is easier to complete 


SLR obtains more useful information 


SLR is more convenient 



Figure 3: 

Responses to the question: 
Do you think the SLR report is 
much better, somewhat better, 
about the same, somewhat 
worse, or much worse than the 
format your graduate school uses 
to analyze applicants' letters of 
recommendation? 



T 


H Much better 

□ Somewhat better 

□ About the same 

□ Somewhat worse 

□ Much worse 
■ Do not know 


0% 10% 20% 30% 40% 
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Figure 4: 

Responses to 

questions about Is a useful summary 
specific features of 
the SLR Report 


Averages of three ratings is useful 


Provides a better understanding of the candidate 


Provides all the info I need 



Participants described their reactions to specific features of the 
SLR Report. (See Figure 4). A majority of respondents strongly 
agreed or somewhat agreed that the SLR Report provided a use- 
ful summary of the evaluators’ assessment and the averages of 
the three evaluators is a useful way to summarize evaluations. 
A smaller majority agreed that the SLR Report gives the reader 
a better understanding of the applicant than does the narrative 
letter and “includes all the information I want to know from the 
letter writers.” 

When asked about the usefulness of the comparative 
norms in the report, 25 percent of the sample rated them as 
very useful, 61 percent as somewhat useful, and 12 percent 
as not useful. Moreover, 70 percent of the sample would like 
to have the SLR report combined with the applicant’s GRE 
Score report. 

The majority of participants did not provide suggestions 
for revisions to the SLR Report. However, the top four sug- 
gestions were: (a) the narrative should be displayed with the 
rating for each quality (four percent of the total sample); (b) 
add more qualities to the SLR (e.g., discipline-specific) (two 
percent); (c) add questions about specific experiences and 
knowledge of the field (four percent); and (d) programs should 
be able to customize the forms (four percent). 


The SLR Adoption Process 

When asked whether they believed that the SLR would be 
widely adopted by graduate schools, 50 percent strongly 
agreed or somewhat agreed while 30 percent strongly dis- 
agreed or somewhat disagreed. Overall, 42 percent of the 
sample believed that it would either be very easy or somewhat 
easy to get their graduate school to switch to the SLR, while 
45 percent believed that adoption would either be very diffi- 
cult or somewhat difficult. When asked whether they thought 
that most faculty members in their department would prefer 
the SLR to current letters of recommendation, 51 percent of 
the participants said yes, 34 percent said no. When asked 
about the likelihood of their schools adopting the SLR in the 
next three years (if it was available today), 60 percent said it 
would be very likely or somewhat likely, and 32 percent said 
it would be very unlikely or somewhat unlikely. 

Of the total sample of respondents, 30 percent said that 
the SLR should replace current letters of recommendation, 
while 51 percent believed that it should supplement current 
letters. Eighteen percent said that it should neither replace 
nor supplement letters. 
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“Respondents were somewhat split in their estimates of the ease and 
likelihood of adopting the SLR into their graduate admission process. 
Clearly, adoption of the SLR will involve a general openness to transforming 
the current recommendation process and also a high-level of cooperation 
and coordination among graduate institutions.” 


Discussion 

Overall, graduate faculty and administrators responded positively 
to the concept, functionality and content of the SLR system, with 
a majority of the respondents indicating that the SLR is better 
than what is currently being used by their institutions. In com- 
parison to narrative letters, a majority of respondenfs agreed that 
the SLR Web site is quicker, easier and more convenient for fhe 
lefter writer. Moreover, a majority agreed that the SLR Report pro- 
vides a useful summary of the evaluators' assessments, reports 
more useful and reliable information about the applicant, and 
ultimately promotes fairer comparisons. 

Further comments suggest that users would prefer other 
applicant materials such as GRE scores to be combined with 
the SLR Report. The SLR could be either a stand-alone service 
or a component of a larger new service for graduate schools. 
The full service approach would be a Web-based applicafion 
service consisting of the application form, transcript summa- 
ry, score reports, personal statements, and the SLR. The SLR 
graphical report might include candidates' GRE test scores 
alongside averaged ratings on each quality. 

Respondents were somewhat split in their estimates of the 
ease and likelihood of adopfing the SLR into their graduate 
admission process. Clearly, adoption of fhe SLR will involve a 
general openness fo fransforming the current recommendation 
process and also a high-level of cooperation and coordination 
among graduate institutions. Future work will need to expand 
on these initial estimates by collaborating with representatives 
from graduafe institutions to discuss not only an appropriate 
strategy for adopfing the SLR, but also to continue to assess 
the full range of pracfical and logistical implications involved. 

Practical Implications of Adopting the SLR 

This initial analysis of faculty and administrator reactions to 
the SLR indicates that the SLR produces information that 
may enhance the current admission process. Flowever, its use 


also introduces several issues. First, quantifying fhe rafings 
of candidafes generates an explicit evaluation of the appli- 
cant. This differs from fhe current practice in which implicit 
statements are embedded in text. The explicitness of the SLR 
ratings raises concern about legal accountability of evalu- 
ators. Currently, applicants can waive their right to access 
letters written on their behalf. This system would apply to the 
SLR, but it is critical that the legal boundaries of fhis waiver 
are established. 

A second issue involves the capability of the SLR to gen- 
erate data that can be used for a variety of research, evalu- 
ative, and applicant selection purposes. Currently, letters of 
candidates not admitted for study are destroyed. Perhaps 
SLR data could be retained so long as all personal identifiers 
are removed, but the potential legal ramifications of doing 
so must be considered. Flowever, even if dafa on candidates 
who are not admitted cannot be retained for legal exposure 
reasons, data on admitted candidates would still be useful 
for comparisons of candidafes with other students in the pro- 
gram and at the institution, to compare across programs and 
institutions, and to further our understanding of predictors of 
graduafe school success. 

A third issue to consider is that the longitudinal data can 
also be used to assess the rating standards of each evalu- 
ator — an option that might prove highly valuable to admis- 
sion committees when assessing the validity of the evalua- 
tions. Flowever, this idea is somewhat controversial given the 
heightened level of accountability it entails for faculty mem- 
bers. Perhaps faculty can indicate on the Web site whether 
they will allow the data to be kept. Again, the ability to retain 
the data of applicants who are not admitted remains uncer- 
tain, and the absence of such data would limit an accurate 
assessment of standards. 

Finally, researchers must consider that it may become nec- 
essary to develop program-specific, or discipline-specific SLR's, 
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“Schools can use 
this information to 
understand how 
to better prepare 
students for higher 
education and 
specific major 
areas.” 


or at a minimum, SLR's for more and less technical graduate 
programs. Focus group participants expressed doubt about a 
consensus of a sufficient list of qualities across departments. 

Future Work to Develop the SLR 

This study was a first step toward the development of the SLR. 
Additional studies could be done to determine the reliability and 
validity of the standardized letter. Researchers could assess the 
validity of the SLR by tracking a group of students longitudi- 
nally through their graduate programs. Data could be collected 
on degree completion, graduate-school grade-point average, fac- 
ulty ratings of student performance, and other graduate school 
outcome variables. These could be used as criteria against which 
to conduct incremental validity studies of the SLR vs. letters of 
recommendation, college GPA, GRE scores, and perhaps other 
predictor variables. 

While a majority of respondents in this sample believed that 
the SLR prototype contained the qualities most important to suc- 
cess in graduate school, validity and reliability data would allow 
an empirical determination about whether these qualities are, 
indeed, the best ones to include on the SLR for graduate admis- 
sion. Moreover, on-going item-level analyses would allow us to 
determine the appropriate structure of the SLR (e.g., do seven 
qualities capture the evaluation better than three?). 

Implications for the College Admission Community 
SLR as a Contribution to Fairness Initiatives 

Students who traditionally score low on standardized tests may 
benefit from the opportunity for evaluators to provide additional 
information about them in a way that is valid and systematic. 
Supplementing standardized tests with valid information on non- 
cognitive qualities that are also relevant to educational outcomes 
should increase prediction and reduce group differences {Sack- 
ett, Schmitt, Ellingson, and Kabin, 2001). This opportunity 
would address college counselors’ and admission professionals’ 
concern for underperforming students. 


SLR as a Research Tool for the Higher Education Community 

Why would schools be willing to participate in the implemen- 
tation of the SLR, once the background research has been 
conducted and the validity has been studied? A “feedback 
loop’’ may provide an incentive for school faculty to participate 
in the use of the SLR. As years pass, SLR data will provide 
information about perceived qualitative traits that predict suc- 
cessful academic outcomes. Schools can use this informa- 
tion to understand how to better prepare students for higher 
education and specific major areas. The SLR data will allow 
admission committees to garner a better sense of the types of 
students who will persist in specific areas of study and institu- 
tions, leading to better matches between students and pro- 
grams, and perhaps therefore increased student performance 
and reduced attrition. 

This study represented a necessary first step toward under- 
standing the viability of introducing the SLR into the graduate 
admission process. Any potential user of the SLR for undergrad- 
uates or specific fields will need to conduct this initial first stage 
of research to determine the interest, functionality, and content 
that is appropriate for their user population. 
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Appendix A 

The top four reasons listed within each response 
category for comparison of SLR to traditional letters 
(the sample is collapsed across graduate faculty, 
HBCU faculty and administrators). 


Why do you say that the SLR Report is much bettefl 

(A/= 

94) 


n 

% 

Detailed list of qualities is more thorough and stimulates 
the writer 

24 

28 

A comprehensive list of qualities 

21 

24 

It is convenient and easy 

13 

15 

Provides more information than standard narrative letters 

11 

13 

Why do you say that the SLR Report is somewhat betterl 

(/V= 

120) 


n 

% 

Detailed list of qualities is more thorough and stimulates 
the writer 

40 

28 

Allows for standardized evaluation/comparison of 
candidates 

21 

15 

Comprehensive list of qualities 

20 

14 

It is convenient and easy 

12 

8 

Why do you say that the SLR Report is about the samel 

{N= 

57) 


n 

% 

Basically, the same information as the narrative form 

46 

61 

Prefer a personalized and tailored letter 

8 

11 

SLR prototype is missing some qualities 

5 

7 

It is convenient and easy 

3 

4 

Why do you say that the SLR Report is somewhat worsel 

{N= 

33) 


n 

% 

Prefer a personalized and tailored letter 

9 

20 

SLR prototype is inconvenient, too long 

8 

18 

SLR prototype has too many items per quality 

7 

16 

SLR prototype is missing some qualities 

6 

13 

Why do you say that the SLR Report is much worsel 

(N-- 

= 8) 


n 

% 

Report de-emphasizes qualitative information 

5 

31 

Ease of completion may lead to a superficial evaluation 

3 

19 

Does not provide for description of specific skills 

3 

19 

SLR prototype is inconvenient, too long 

2 

13 


Note: Not all participants provided an explanation for their responses. 


Appendix B 

The top four reasons listed within each response category 
for comparison of the SLR Report to the format used at 
their graduate schools (the sample is collapsed across 
graduate faculty, HBCU faculty and administrators). 


Why do you say that the SLR Report is much betterl 

(/V= 

125) 


n 

% 

Currently do not have a summary form 

24 

22 

Provides a useful summary 

20 

18 

Ability to compare applicants in a fairer way 

19 

17 

Comparison norms 

17 

15 

Why do you say that the SLR Report is somewhat betterl 

{N= 

120) 


n 

% 

Evaluation is more comprehensive than current one 

21 

19 

Standardized, uniform information 

18 

16 

Currently do not have a summary form 

17 

15 

Ability to compare applicants in a fairer way 

15 

14 

Why do you say that the SLR Report is about the samel 

(/V= 

57) 


n 

% 

Use a similar form 

25 

57 

Report de-emphasizes qualitative information 

5 

11 

Not the information Iwant 

4 

9 

Our current process is better and more informative 

3 

7 

Why do you say that the SLR Report is somewhat worsel 

(/V= 

33) 


n 

% 

Report de-emphasizes qualitative information 

7 

32 

Not the information Iwant 

6 

27 

Hard to understand the report 

4 

18 

Our current process is better and more informative 

3 

14 

Why do you say that the SLR Report is much worsel 

(/V= 

= 8) 


n 

% 

Report de-emphasizes qualitative information 

3 

43 

Not the information Iwant 

2 

29 

Averages conceal important individual rating information 

2 

29 

Hard to understand the report 

1 

14 


Note: Not all participants provided an explanation for their responses. 
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